Papers with compression approaches
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models (2022.emnlp-main)
Copied to clipboard
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, Dan Alistarh
| Challenge: | Pre-trained Transformer models provide robust language representations which can be specialized on various tasks. |
| Approach: | They propose an efficient pruning method based on approximate second-order information that allows pruning weight blocks to be used for pruning. |
| Outcome: | The proposed method is the first to be applied at the BERT scale and significantly pushes the boundaries of the current sparse models with respect to all metrics: model size, inference speed and task accuracy. |
Adaptive Parameter Compression for Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Adaptive parameter compression is a new approach to improve NLP models . the current algorithm is based on a single parameter, but it is not scalable. |
| Approach: | They propose a hardware-independent compression strategy that extends the weight-squeezing approach by introducing compression biases and weights. |
| Outcome: | The proposed compression strategy outperforms DistilBERT base models while being significantly more efficient. |
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression approaches prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Approach: | They propose a structure-aware KV cache compression framework that prioritizes tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Outcome: | The proposed framework preserves long-range dependencies and retrieval robustness. |
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression approaches remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. |
| Approach: | They propose a structured **T**rapezoidal **E**xpert **N**euron **P**running framework that uses a trapezoidal pattern to identify and retain important experts while applying expert neuron pruning (ENP) to less important experts. |
| Outcome: | The proposed framework outperforms the full-parameter model by 10% on code generation tasks under a sparse activation of experts and a 40% routing sparsity. |